gh-126105: Fix crash in `ast` module, when `._fields` is deleted #126115

sobolevn · 2024-10-29T08:55:53Z

This PR changes the logic a bit, but it looks like a correct thing to me.
Previously we were handling NULL of fields case for remaining_fields

cpython/Python/Python-ast.c

Lines 5089 to 5098 in 9b14083

    
           if (fields) { 
        
               numfields = PySequence_Size(fields); 
        
               if (numfields == -1) { 
        
                   goto cleanup; 
        
               } 
        
               remaining_fields = PySet_New(fields); 
        
           } 
        
           else { 
        
               remaining_fields = PySet_New(NULL); 
        
           }

But, we didn't really handle the fields itself.
Further calls assume that fields cannot be NULL.

The other way of fixing this is to add a default for fields like:

    if (fields == NULL) {
        fields = PyList_New(0);
        if (fields == NULL) {
            goto cleanup;
        }
    }

In this case ast.AST(arg=1) with no _fields will produce:

DeprecationWarning: ast.AST.__init__ got an unexpected keyword argument 'arg'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.

It does not seem correct to me, because _fields is a part of our API. Since AST objects are mostly internal, there are no real cases of missing _fields.

Issue: Segmentation fault after deleting __fields__ attribute from ast.AST #126105

picnixz

I'm wondering why we have:

if (PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields) < 0) {

This could be because some AST nodes do not have a _fields attribute but then, why don't we simply change it to an eager PyObject_GetAttr?

Misc/NEWS.d/next/Library/2024-10-29-11-45-44.gh-issue-126105.cOL-R6.rst

Parser/asdl_c.py

sobolevn · 2024-10-29T09:44:20Z

Docs give me more confidence: https://docs.python.org/3/library/ast.html#ast.AST._fields

Each concrete class has an attribute _fields which gives the names of all child nodes.

sobolevn · 2024-10-29T10:26:41Z

I don't think that idle_tests are related to this change. But, I will take a look.

Eclips4

LGTM. Thanks!

Lib/test/test_ast/test_ast.py

JelleZijlstra · 2024-10-29T14:23:35Z

Parser/asdl_c.py

    Py_ssize_t i, numfields = 0;
    int res = -1;
    PyObject *key, *value, *fields, *attributes = NULL, *remaining_fields = NULL;
-    if (PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields) < 0) {


A simpler fix would have been this, right?

if (PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields) <= 0) {

However, your fix is better.

miss-islington-app · 2024-10-29T15:42:52Z

Thanks @sobolevn for the PR, and @Eclips4 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

pythonGH-126115) Previously, if the `ast.AST._fields` attribute was deleted, attempts to create a new `as`t node would crash due to the assumption that `_fields` always had a non-NULL value. Now it has been fixed by adding an extra check to ensure that `_fields` does not have a NULL value (this can happen when you manually remove `_fields` attribute). (cherry picked from commit b2eaa75) Co-authored-by: sobolevn <[email protected]>

miss-islington-app · 2024-10-29T15:43:02Z

Sorry, @sobolevn and @Eclips4, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker b2eaa75b176e07730215d76d8dce4d63fb493391 3.12

bedevere-app · 2024-10-29T15:43:05Z

GH-126130 is a backport of this pull request to the 3.13 branch.

Eclips4 · 2024-10-29T15:49:33Z

I'll take care of backport to 3.12.

… deleted (pythonGH-126115) Previously, if the `ast.AST._fields` attribute was deleted, attempts to create a new `as`t node would crash due to the assumption that `_fields` always had a non-NULL value. Now it has been fixed by adding an extra check to ensure that `_fields` does not have a NULL value (this can happen when you manually remove `_fields` attribute). (cherry picked from commit b2eaa75) Co-authored-by: sobolevn <[email protected]>

bedevere-app · 2024-10-29T15:59:56Z

GH-126132 is a backport of this pull request to the 3.12 branch.

…ed (GH-126115) (#126130) gh-126105: Fix crash in `ast` module, when `._fields` is deleted (GH-126115) Previously, if the `ast.AST._fields` attribute was deleted, attempts to create a new `as`t node would crash due to the assumption that `_fields` always had a non-NULL value. Now it has been fixed by adding an extra check to ensure that `_fields` does not have a NULL value (this can happen when you manually remove `_fields` attribute). (cherry picked from commit b2eaa75) Co-authored-by: sobolevn <[email protected]>

sobolevn · 2024-10-29T16:46:49Z

@Eclips4 thank you for your help!
@JelleZijlstra @picnixz thanks for the quick reviews!

#126132) [3.12] gh-126105: Fix crash in `ast` module, when `._fields` is deleted (GH-126115) Previously, if the `ast.AST._fields` attribute was deleted, attempts to create a new `as`t node would crash due to the assumption that `_fields` always had a non-NULL value. Now it has been fixed by adding an extra check to ensure that `_fields` does not have a NULL value (this can happen when you manually remove `_fields` attribute). (cherry picked from commit b2eaa75) Co-authored-by: sobolevn <[email protected]>

python#126115) Previously, if the `ast.AST._fields` attribute was deleted, attempts to create a new `as`t node would crash due to the assumption that `_fields` always had a non-NULL value. Now it has been fixed by adding an extra check to ensure that `_fields` does not have a NULL value (this can happen when you manually remove `_fields` attribute).

pythongh-126105: Fix crash in ast module, when ._fields is deleted

f64b60b

sobolevn requested review from JelleZijlstra and picnixz October 29, 2024 08:55

sobolevn requested review from Eclips4 and isidentical as code owners October 29, 2024 08:55

bedevere-app bot added the awaiting core review label Oct 29, 2024

bedevere-app bot mentioned this pull request Oct 29, 2024

Segmentation fault after deleting __fields__ attribute from ast.AST #126105

Closed

picnixz reviewed Oct 29, 2024

View reviewed changes

Misc/NEWS.d/next/Library/2024-10-29-11-45-44.gh-issue-126105.cOL-R6.rst Outdated Show resolved Hide resolved

Parser/asdl_c.py Outdated Show resolved Hide resolved

Address review

980aa20

sobolevn force-pushed the issue-126105 branch from 32f5eed to 980aa20 Compare October 29, 2024 09:42

Eclips4 approved these changes Oct 29, 2024

View reviewed changes

Lib/test/test_ast/test_ast.py Show resolved Hide resolved

bedevere-app bot added awaiting merge and removed awaiting core review labels Oct 29, 2024

Eclips4 added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Oct 29, 2024

JelleZijlstra reviewed Oct 29, 2024

View reviewed changes

JelleZijlstra approved these changes Oct 29, 2024

View reviewed changes

Eclips4 merged commit b2eaa75 into python:main Oct 29, 2024
47 checks passed

bedevere-app bot removed the awaiting merge label Oct 29, 2024

miss-islington-app bot assigned Eclips4 Oct 29, 2024

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Oct 29, 2024

Eclips4 mentioned this pull request Oct 29, 2024

[3.12] gh-126105: Fix crash in ast module, when ._fields is delet… #126132

Merged

bedevere-app bot removed the needs backport to 3.12 only security fixes label Oct 29, 2024

	if (fields) {
	numfields = PySequence_Size(fields);
	if (numfields == -1) {
	goto cleanup;
	}
	remaining_fields = PySet_New(fields);
	}
	else {
	remaining_fields = PySet_New(NULL);
	}

Uh oh!

gh-126105: Fix crash in ast module, when ._fields is deleted #126115

gh-126105: Fix crash in ast module, when ._fields is deleted #126115

Uh oh!

Conversation

sobolevn commented Oct 29, 2024 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sobolevn commented Oct 29, 2024

Uh oh!

sobolevn commented Oct 29, 2024

Uh oh!

Eclips4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JelleZijlstra Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Oct 29, 2024

Uh oh!

miss-islington-app bot commented Oct 29, 2024

Uh oh!

bedevere-app bot commented Oct 29, 2024

Uh oh!

Eclips4 commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Oct 29, 2024

Uh oh!

sobolevn commented Oct 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-126105: Fix crash in `ast` module, when `._fields` is deleted #126115

gh-126105: Fix crash in `ast` module, when `._fields` is deleted #126115

sobolevn commented Oct 29, 2024 •

edited by bedevere-app bot

Loading

Eclips4 commented Oct 29, 2024 •

edited

Loading